State Noise Effects on the Stochastic Gradient Descent Optimization Method for Echo State Networks with Leaky Integrator Neurons

Executive Summary Echo state networks (ESNs) are a novel approach to modeling the nonlinear dynamical systems that abound in the sciences and engineering. They employ artificial recurrent neural networks in a way that has been independently proposed as a learning mechanism in biological brains and lead to a fast and simple algorithm for supervised training. ESNs are controlled by several global learning parameters that may be optimized using techniques such as the stochastic gradient descent method in order to improve the performance of the network. The research is an empirical study of the stochastic gradient descent method leading to a better understanding of that optimization approach, its strengths and limitations, which results in more efficient applications of ESNs in all basic tasks of signal processing and control, including time series prediction, inverse modeling, pattern generation, event detection and classification, modeling distributions of stochastic processes, filtering and nonlinear control.